Method for automatically generating waypoints for imaging shelves within a store

ABSTRACT

One variation of a method for automatically generating waypoints for imaging shelves within a store includes: dispatching a robotic system to autonomously generating a map of a floor space within the store; accessing an architectural metaspace defining target locations and addresses of the set of shelving structures within the store; distorting the architectural metaspace into alignment with the map to generate a normalized metaspace representing real locations and addresses of the set of shelving structures in the store; defining a set of waypoints distributed longitudinally along and offset laterally from a first shelving structure represented in the normalized metaspace based on a known position of an optical sensor in the robotic system; and dispatching the robotic system to record optical data while occupying the set of waypoint optv-mo3 in:#finance s during an imaging routine.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/513,228, filed on 16 Jul. 2019, which is a continuation of U.S.patent application Ser. No. 15/600,556, filed on 19 May 2017, whichclaims the benefit of U.S. Provisional Application No. 62/339,047, filedon 19 May 2016, U.S. Provisional Application No. 62/339,039, filed on 19May 2016, and U.S. Provisional Application No. 62/339,045, filed on 19May 2016, each of which is incorporated in its entirety by thisreference.

The application is related to U.S. patent application Ser. No.15/347,689, filed on 9 Nov. 2016, and to U.S. patent application Ser.No. 15/600,527, filed on 19 May 2017, each of which is incorporated inits entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the field of stock tracking and morespecifically to a new and useful method for automatically generatingwaypoints for imaging shelves within a store in the field of stocktracking.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a method;

FIG. 2 is a flowchart representation of one variation of the method;

FIG. 3 is a flowchart representation of one variation of the method;

FIG. 4 is a flowchart representation of one variation of the method;

FIG. 5 is a flowchart representation of one variation of the method; and

FIG. 6 is a flowchart representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Method

As shown in FIG. 1, a method S100 for automatically generating waypointsfor imaging shelves within a store includes: dispatching a roboticsystem to autonomously map a floor space within a store in Block S110;receiving a map of the floor space from the robotic system in BlockS120; accessing an architectural metaspace defining locations ofshelving structures and addresses of shelving structures within thestore in Block S130; generating a normalized metaspace by aligningfeatures in the architectural metaspace to features in the map in BlockS140; defining a coordinate system for the store within the normalizedmetaspace in Block S150; defining a set of waypoints within thenormalized metaspace based on positions of shelving structures withinthe normalized metaspace and imaging capabilities of the robotic systemin Block S160, each waypoint including a location and orientationrelative to the coordinate system; labeling each waypoint in the set ofwaypoints with an address of a shelving structure proximal the waypointin Block S170; and uploading the set of waypoints to the robotic systemfor execution in a subsequent imaging routine in Block S180.

The method S100 can also include: generating the architectural metaspacein Block S130; labeling shelving structures in the normalized metaspacewith addresses of corresponding shelving segments and shelves from a setof preset elevation maps—or “scenes” stored in a planograms of thestore—based on shelving structure addresses in the normalized metaspacein Block S142; and labeling a waypoint with an address of a camera inthe robotic system scheduled to capture an image and addresses of ashelving structure, a shelving segment, and a shelf anticipated to be ina field of view of the camera when the robotic system is proximal thelocation and orientation within the store defined in the waypoint inBlock S170.

One variation of the method S100 includes: dispatching a robotic systemto autonomously collect map data of a floor space within the storeduring a mapping routine in Block S110; generating a map of the floorspace from map data collected by the robotic system during the mappingroutine in Block S120; accessing an architectural metaspace definingtarget locations and addresses of the set of shelving structures withinthe store in Block S130; distorting features in the architecturalmetaspace into alignment with like features in the map to generate anormalized metaspace representing real locations and addresses of theset of shelving structures in the store in Block S140; defining acoordinate system within the normalized metaspace in Block S150;defining a set of waypoints within the normalized metaspace relative tothe coordinate system in Block S160, the set of waypoints including afirst subset of waypoints distributed longitudinally along and offsetlaterally from a real location of a first shelving structure, in the setof shelving structures, represented in the normalized metaspace, eachwaypoint in the first subset of waypoints defining an orientationrelative to the coordinate system based on a known position of anoptical sensor in the robotic system and the real location of the firstshelving structure in the normalized metaspace; and dispatching therobotic system to record optical data while occupying each waypoint inthe first subset of waypoints during an imaging routine in Block S180.

Another variation of the method S100 includes: dispatching a roboticsystem to autonomously map a floor space within the store in Block S110;receiving a map of the floor space from the robotic system in BlockS120; accessing an architectural metaspace defining locations ofshelving structures and addresses of shelving structures within thestore in Block S130; generating a normalized metaspace by aligningfeatures in the architectural metaspace to features in the map in BlockS140; defining a coordinate system for the store within the normalizedmetaspace in Block S150; defining a set of waypoints within thenormalized metaspace based on positions of shelving structures withinthe normalized metaspace and imaging capabilities of the robotic systemin Block S160, each waypoint including a location and orientationrelative to the coordinate system; labeling each waypoint in the set ofwaypoints with an address of a shelving structure proximal the waypointin Block S170; and uploading the set of waypoints to the robotic systemfor execution in a subsequent imaging routine in Block S180. In thisvariation, the method S100 can also include: accessing a first imageincluding visual data recorded by the robotic system when occupying afirst waypoint in the set of waypoints during the imaging routine;detecting a first shelf in the first image; determining an address ofthe first shelf based on a first address of a first shelving structureassociated with the first waypoint and a vertical position of the firstshelf within the first image; based on the address of the first shelf,retrieving a first list of products assigned to the first shelf by aplanogram of the store; retrieving a first set of template images from adatabase of template images, each template image in the first set oftemplate images including visual features of a product specified in thefirst list of products; extracting a first set of features from a firstregion of the first image adjacent the first shelf; confirming presenceof a unit of a first product, in the first list of products, on thefirst shelf in response to correlation between features in the first setof features and features in a first template image, in the first set oftemplate images, representing the first product; determining that a unitof the first product is improperly stocked on the first shelf inresponse to deviation between features in the first set of features andfeatures in the first template image; and, in response to determiningthat the unit of the first product is improperly stocked on the firstshelf, generating a first restocking prompt for the first product on thefirst shelf in Block S190.

2. Applications

Generally, the method S100 can be executed by a computer system toautomatically generate a set of waypoints for a store based on realmapping data collected by a robotic system dispatched to the store,based on stocking data stored in elevation maps or “scenes” of discreteshelving structures defined in a saved planogram or other productposition database, and based on imaging capabilities of the roboticsystem. Upon receipt of these waypoints, the robotic system cansequentially navigate to locations defined in these waypoints, captureone or more images at each of these waypoints, and return these imagesto the computer system. The computer system can then automaticallytransform these images into a graph, table, or map of positions ofproducts on shelves throughout the store and deviations from productpresentation specifications defined for the store, as shown in FIG. 2.

In particular, the method S100 can be implemented by a local or remotecomputer system to automatically define and tag waypoints for a storebased on a combination of data collected by the robotic system andpreexisting data defining the layout of shelves and/or products withinthe store. For example, for a small store stocked with 1,000 uniqueproducts (i.e., 1,000 unique stock keeping units, or “SKUs”), the systemcan automatically generate 20 unique waypoints for a robotic systemnavigating through the store to collect images of products stocked onshelves in the store. In this example, for the robotic system thatincludes four cameras, as described below, the robotic system cancollect up to 80 images of shelves in the small store (i.e., four imagesper waypoint) during a imaging routine, and the computer system cantransform these images into a map of products within the small store anddeviations from product presentation requirements predefined for thestore. However, for a store of moderate size stocked with 25,000 uniqueproducts (i.e., 25,000 unique SKUs), the system can automaticallygenerate 200 unique waypoints for the store, and the robotic systemincluding four cameras can capture up to 800 images of shelves withinthe store. Furthermore, for a large store stocked with 150,000 uniqueproducts, the system can automatically generate 500 unique waypoints forthe store, and the robotic system including four cameras can capture upto 2,000 images of shelves within the store.

For example, the robotic system can be placed within a retail store (orwarehouse, etc.), and the remote computer system (e.g., a remote serverconnected to the robotic system via the Internet) can dispatch therobotic system to collect images at waypoints throughout the retailstore outside of store hours, such as every night beginning at 1 AM. Therobotic system can thus complete a imaging routine before the retailstore opens a few hours later. Furthermore, the system can processimages collected during the imaging routine to generate a graph, map, ortable of current product placement on shelves in the store and/or togenerate a task list of misplaced or misoriented products to correct,and the system can present this graph, map, table, and/or task list toemployees upon their arrival at the retail store the next morning beforethe retail store opens. Alternatively, the remote computer system candispatch the robotic system to waypoints within the store during storehours and can process images received from the robotic systemsubstantially in real-time to generate current such graphs, maps,tables, and/or task lists in near-real-time.

The method S100 can be executed when the robotic system is provisionedto a new store or retrofit to an existing store. For example, thecomputer system can aggregate an architectural plan for the store (or ageneric store layout assigned to the store) and a set of elevation mapsfor the store into an architectural metaspace defining anticipatedpositions of shelving structures, shelving segments, and/or shelves—ofknown addresses—within the store. However, because real positions ofshelving structures and shelving segments (and aisles, walls, columns,etc.) in the store may differ from positions represented in thearchitectural metaspace, the computer system can collect real floorlayout data for the store through the robotic system and align featureswithin the architectural metaspace to these real floor layout data. Oncethe architectural metaspace is transformed into a normalized metaspacerepresentative of real positions of shelving structures and shelvingsegments, etc. within the store, the system can generate a set ofwaypoints that each define a target location and orientation for therobotic system and address one or more cameras within the robotic systemthrough which the robotic system is to record photographic images whenoccupying the waypoint. The method S100 can be similarly executed aftera store is renovated or after permanent or temporary shelving structuresare moved within the store in order to recalculate waypoints for thestore.

Therefore, the system can implement Blocks of the method S100 to:collect map data representing true locations of objects within a store;access an architectural (or similar) plan specifying target or plannedlocations of these objects within the store and containing links to aplanogram of the store; and merge the map and the architectural plan tocreate a normalized metaspace that represents accurate locations ofknown, addressed objects (e.g., shelving structures, shelving segments,etc.) within the store. The system can then define a set of waypointsthat—when occupied by the robotic system to within a threshold linearand angular tolerance—locate shelves, shelving segments, and shelvingstructures throughout the store in regular, repeatable positions in thefields of view of cameras integrated into the robotic system. The systemcan thus: automatically generate a set of waypoints linked to elevationmaps of shelving structures in a planogram of the store; while alsocompensating for deviations from an original layout plan for the storeand achieving a high degree of precision (i.e., accuracy andrepeatability) in images of shelves, shelving segments, and shelvingstructures recorded by the robotic system when executing an imagingroutine; and while further reducing or eliminating human supervision orhuman labeling of objects represented in the map (e.g., entry ofaddresses of shelving structures represented in the map). By executingthe method S100 to thus define a set of waypoints for the store withlimited or no human supervision, the system can achieve relatively rapidonboarding and provisioning of the robotic system to the store, achievea high quality of images of shelves, shelving segments, and shelvingstructures, etc. recorded by the robotic system, and achieve a highdegree of confidence in the accuracy of a stocking condition of thestore based on data extracted from these images.

3. Robotic System

A robotic system executes Blocks S110 and S112 of the method S100 tonavigate to a waypoint and to capture images of shelves in the store.Generally, the robotic system can define a network-enabled mobile robotthat can autonomously traverse a store, capture images of shelves withinthe store, and upload those images to a remote computer system foranalysis.

In one implementation, the robotic system defines an autonomous imagingvehicle including: a base; a drive system (e.g., a pair of two drivenwheels and two swiveling castors) arranged in the base; a power supply(e.g., an electric battery); a set of mapping sensors (e.g., fore andaft scanning LIDAR systems); a processor that transforms data collectedby the mapping sensors into two- or three-dimensional maps of a spacearound the robotic system; a mast extending vertically from the base; aset of cameras arranged on the mast; a geospatial position sensor (e.g.,a GPS sensor); and a wireless communication module that downloadswaypoints and a master map of a store from a remote computer system(e.g., a remote server) and that uploads photographic images captured bythe camera and maps generated by the processor to the remote computersystem, as shown in FIG. 3. In this implementation, the robotic systemcan include cameras mounted statically to the mast, such as twovertically offset cameras on a left side of the mast and two verticallyoffset cameras on the right side of mast, as shown in FIG. 3. Therobotic system can additionally or alternatively include articulablecameras, such as: one camera on the left side of the mast and supportedby a first vertical scanning actuator; and one camera on the right sideof the mast and supported by a second vertical scanning actuator. Therobotic system can also include a zoom lens, a wide-angle lens, or anyother type of lens on each camera. However, the robotic system candefine any other form and can include any other subsystems or elementssupporting autonomous navigating and image capture throughout a storeenvironment.

The system can also include multiple robotic systems placed in a singlestore and configured to cooperate to image shelves within the store. Forexample, two robotic systems can be placed in a large single-floorretail and can cooperate to collect images of all shelves in the storewithin a threshold period of time (e.g., within one hour). In anotherexample, one robotic system can be placed on each floor of a multi-floorstore, and each robotic system can each collect images of shelves on itscorresponding floor. The remote computer system can then aggregateimages captured by multiple robotic systems placed in one store togenerate a graph, map, table, and/or tasklist of properly- andimproperly-stocked slots within the store.

4. Hierarchy and Terms

A “product facing” is referred to herein as a side of a product (e.g.,of a particular SKU or other product identifier) designated for a slot.A “planogram” is referred to herein as a graphical representation ofmultiple product facings across each of multiple shelving structureswithin a store (e.g., across an entire store). Product identification,placement, and orientation data recorded visually in a planogram can bealso be recorded in a corresponding textual product placementspreadsheet, slot index, or other store database (hereinafter a “productplacement database”).

A “slot” is referred to herein as a section of a shelf designated foroccupation by one product facing, including a row of one or more unitsof a product. A “shelf” is reference herein as one lateral surface(e.g., one four-foot-wide horizontal surface) spanning one or moreslots. A “shelving segment” is referred to herein as one column of ashelving structure, including one or more shelves. A “shelvingstructure” is referred to herein as a row of one or more shelvingsegments. An “aisle” is referred to herein as a thoroughfare between twoopposing shelving structures. A “store” is referred to herein as a(static or mobile) facility containing one or more shelving structuresand one or more aisles.

A “product” is referred to herein as a type of packaged good associatedwith a particular product identifier (e.g., a SKU). A “unit” or “productunit” is referred to herein as an instance of a product, such as onepackaged article in a series of packaged articles associated with oneSKU value.

The method S100 is described herein as executed by a remote computersystem (e.g., a remote server, hereinafter a “computer system”).However, Blocks of the method S100 can be executed by one or morerobotic systems placed in a retail space (or store, warehouse, etc.), bya local computer system, or by any other computer system—hereinafter a“system.”

Furthermore, Blocks of the method S100 are described below as executedby the system to identify products stocked on open shelves on shelvingstructures within a store. However, the system can implement similarmethods and techniques to identify products stocked in cubbies, in arefrigeration unit, on a wall rack, in a freestanding floor rack, on atable, or on or in any other product organizer in a retail space.

5. Robotic System Provisioning

Block S110 of the method S100 recites dispatching a robotic system toautonomously collect map data of a floor space within the store during afirst mapping routine. Generally, in Block S110, the system triggers arobotic system to autonomously navigate through the store and togenerate a 2D map of the floor space within the store (and/or togenerate a 3D map of the interior volume of the store).

In one implementation, once the robotic system is powered on in a storefollowing initial placement in the store, the system can serve a promptto the robotic system in Block S110, such as via a computer network(e.g., the Internet), to initiate a mapping routine. For example, therobotic system can immediately initiate a mapping routine upon receiptof the prompt from the system, or the robotic system can queue themapping routine for a subsequent scheduled scan time, such as at 2 AMwhen the store is closed or (substantially) unoccupied. Alternatively,the robotic system can automatically initiate a mapping routine whenfirst powered on within the store or in response to a manual input froma user, such as manual selection of a “Start” button on the roboticsystem or within an operator portal executing on a mobile computingdevice and affiliated with the store or the robotic system.

In another implementation, the system dispatches the robotic system toexecute a first mapping routine within the store following initialdelivery of the robotic system to the store and in response to receiptof confirmation of a preferred stocking condition currently present inthe store. For example, in preparation for autonomous generation of anew planogram for the store, associates of the store can manually stockeach slot, shelf, shelving segment, shelving structure, cubby,refrigeration unit, wall rack, freestanding floor rack, and/or table,etc. throughout the store with preferred numbers and orientations ofvarious products; once the robotic system is provisioned to the storeand connected to a local wireless network within the store, a humanoperator (e.g., a manager of the store or entity affiliated with therobotic system) can access an operator portal through a computing device(e.g., a desktop computer or smartphone, etc.), confirm the currentstate of the store fulfills a (approximately) desired stockingcondition, and then select a function to generate a new planogram of thestore according to the current stocking condition. Upon receipt ofconfirmation of the desired stocking condition and the new planogramfunction, the system can initiate execution of the method S100 by firstdispatching the robotic system to collect map data of the store and/orto generate a map of the store in Block S110. However, the system canimplement any other method or technique in Block S110 to trigger therobotic system to execute a mapping routine.

In yet another implementation, the system executes Block S110 inresponse to indication of a change to the layout of the store enteredthrough an operator portal, such as by a manager of the store through aweb browser or native application executing on a desktop computer,mobile computing device, or other device. In particular, followingrearrangement of all or a subset of shelving structures, placement ofnew shelving structures, and removal of existing shelvingstructures—such as intransient (e.g., fixed-infrastructure) shelvingstructures and/or temporary shelving structures (e.g., freestandingcardboard popups)—the system can execute Block S110 to initiategeneration of a new normalized metaspace and new set of waypoints thataccommodate changes to the layout of the store. Therefore, in thisimplementation, the system can: generate a normalized metaspace thatrepresents such a change to the store layout; and then define a set ofwaypoints according to this change to the store layout represented inthe normalized metaspace.

Once triggered to execute a mapping routine by the system (e.g., thecomputer system) in Block S110, the robotic system can autonomouslynavigate through the store while recording distance data through itsintegrated mapping sensors and generating a 2D (or 3D) map of thephysical interior space of the store. For example, the robotic systemcan implement simultaneous localization and mapping (or “SLAM”)techniques to construct and update a map of an unknown environmentwithin the store while also tracking its location within the map basedon LIDAR-based distance scans collected throughout the mapping routine.However, the robotic system can implement any other methods ortechniques to generate a 2D map of the floor space within the store (orto generate a 3D map of the interior of the store).

Alternatively, the robotic system can collect raw map data during amapping routine and upload these data to the system (e.g., a remoteserver, such as via the Internet) in real-time during the mappingroutine or upon conclusion of the mapping routine. The system can thenimplement similar methods and techniques to reconstruct these raw mapdata into a 2D floor plan or 3D map (e.g., a point cloud) of the store(hereinafter a “floor map”) remotely from the robotic system, asdescribed below.

5.1 Multiple Robotic Systems

In one variation, the system includes multiple robotic systems that areprovisioned to a single store, cooperate to collect map data during amapping routine and cooperate to collect images during an imagingroutine, such as by autonomously executing assigned subsets of waypointsdefined for the store. For example, two robotic systems can be placed ina large single-floor retail store and can cooperate to collect map dataand images of all shelves in the store within mapping and imagingroutines of limited duration (e.g., less than one hour). In anotherexample, one robotic system can be placed on each floor of a multi-floorstore, and each robotic system placed in the store can collect map dataand images of shelves on its assigned floor. Following mapping routinesby these robotic systems within the store, the system can collect mapfrom these robotic systems in Block S120 and then implement methods andtechniques described herein to generate one set of waypoints per roboticsystem provisioned to the store

6. Real Floor Map

Block S120 of the method S100 recites receiving a map of the floor spacefrom the robotic system. Generally, in Block S120, the system functionsto collect a 2D map of the floor space of the store (or a 3D map orpoint cloud of the interior of the store) from the robotic systemfollowing completion of the mapping routine described above. Forexample, upon completion of the mapping routine, the robotic system canupload the map to the system via a local Internet-connected wirelessrouter or via a cellular network. However, the system can collect a mapof real locations of obstacles and structures within the store in anyother way and in any other format.

In one implementation shown in FIG. 1, the robotic system generates a 2Dpoint cloud of the store at one horizontal plane offset above the floorof the store (e.g., two inches above the floor of the store) and uploadsthis point cloud to the system in Block S120. In this implementation,the system can then implement line extraction techniques to transformthe 2D point cloud into a vectorized 2D line map representing real(e.g., dimensionally-accurate) positions and external dimensions ofstructures arranged on the floor throughout the store. The system canthen implement pattern matching, structure recognition, templatematching, and/or other computer vision techniques to identify large,discrete, (approximately) rectilinear regions in the vectorized 2D linemap as shelving structures (and/or other storage elements) in the storeand then label the vectorized line map (hereinafter a “floor map”)accordingly in Block S120.

In one example, the system can: label a discrete rectangular structureexhibiting a maximum horizontal dimension greater than one meter andexhibiting an aspect ratio greater than 2:1 as a shelving structure;label a discrete rectangular structure exhibiting a maximum horizontaldimension greater than one meter and exhibiting an aspect ratio lessthan 2:1 as an open table; label a discrete rectangular structureexhibiting a maximum horizontal dimension less than one meter andexhibiting an aspect ratio less than 2:1 as a freestanding popup unit;label a discrete amorphous structure exhibiting a maximum horizontaldimension less than one meter and exhibiting an aspect ratio less than2:1 as a freestanding floor unit; etc. In another example, the systemcan: access a database of standard plan dimensions (e.g., length andwidth) and geometries (e.g., rectangular) of shelving structures,checkout lanes, refrigeration units, etc. common to retail settings;extract dimensions and geometries of structures on the floor of thespace from the 2D line map; and compare these structure dimensions andgeometries to standard plan and geometry definitions stored in thedatabase to identify and label select structures represented in the 2Dline map as shelving structures.

In another example, the system can implement edge detection techniquesto scan a single horizontal plane in the floor map for ˜90° corners,identify a closed region in the floor map bounded by a set of four ˜90°corners as a shelving structure, and identify an open area between twosuch shelving structures as an aisle; the system can then populate thefloor map with labels for shelving structures and aisles accordingly.However, the system can implement template matching, edge detection,pattern matching, pattern recognition, optical character recognition,color recognition, content-based image retrieval, pose estimation, codereading, shape recognition, and/or any other suitable method orprocessing technique to identify features in the floor map and tocorrelate these features with one or more shelving structures within thestore.

6.1 Supervised Shelving Structure Detection

In the foregoing implementation, once the system identifies shelvingstructures represented in a floor map of the store, the system canprompt a human operator (e.g., a robotic system operator, a manager ofthe store) to confirm shelving structure labels autonomously applied tothe floor map. For example, the system can serve a visual form of thefloor map to the operator through the operator portal executing on theoperator's mobile computing device or desktop computer and rendershelving structure labels over the visual form of the floor map; theoperator can then manually review and adjust these shelving structurelabels, thereby providing supervision to the system in identifyingshelving structures throughout the store, such as prior to generation ofwaypoints for an upcoming imaging routine. Alternatively, the system cantransform map data received from the robotic system into a vectorizedfloor map (e.g., a vectorized 2D or 3D line map) of the store, identifydiscrete structures arranged on the floor of the store within the floormap, and serve this floor map with discrete structures highlightedwithin the floor map to the operator portal; the operator can thenmanually confirm that structures highlighted in the floor map representshelving structures through the operator portal. However, the system caninterface with a human operator through the operator portal in any otherway to collect information confirming identification of shelvingstructures or identifying shelving structures directly in the floor mapof the store.

However, the system can implement any other method or technique totransform raw map data (or a vectorized 2D or 3D line map) collected bythe robotic system into a labeled floor map identifying known locationsof shelving structures (and/or other storage elements) within the store,such as relative to a coordinate system described below.

7. Architectural Metaspace

Block S130 of the method S100 recites accessing an architecturalmetaspace defining locations of shelving structures and addresses ofshelving structures within the store. Generally, in Block S130, thesystem retrieves an existing architectural metaspace that defines aplanned location of shelving structures within the store and includesshelving structure addresses or other identifiers that link theseshelving structures to preexisting elevation maps of shelving structuresthroughout the store, wherein these preexisting elevation maps togetherdefine a planogram of the store.

7.1 Architectural Plan

In one implementation, the system generates the architectural metaspacefor the store in Block S130 by linking (or merging) an existingarchitectural plan for the store and preexisting shelving structureelevation maps for the store, as shown in FIG. 1. In one example, a user(e.g., an employee of the store, a robotic system operator) uploads adigital copy of a 2D architectural plan of the store—such as includingpositions of walls, shelves, columns, doors, etc. within thestore—through an operator portal executing with a browser or nativeapplication running on a computing device (e.g., a desktop computer). Inthis example, the system can access a digital scan of paper blueprintsuploaded by the user and then vectorize this digital scan. In anotherexample, the system can retrieve an existing vectorized architecturalplan for the store from an Internet link—supplied by the user throughthe operator portal—to an online data store.

Alternatively, the system can retrieve a standardized store layout(e.g., from a set of standardized store layouts for a particular chainof stores), generally defining an architectural layout of shelvingstructures, kiosks, checkout counters, doors, etc. within the store. Inthis implementation, the system can prompt the user to identify areaswithin the standardized store layout that differ significantly from areal layout of the store, such as missing or added shelving structures,placement of structural columns, etc.; and the system can update thestandardized store layout for the store accordingly.

The system can then implement template matching, edge detection, patternmatching, pattern recognition, optical character recognition, colorrecognition, content-based image retrieval, pose estimation, codereading, shape recognition, and/or any other suitable method orprocessing technique to identify features in the vectorizedarchitectural plan (or vectorized standardized store layout) as shelvingstructures, such as described above for the map of the store.(Hereinafter, such features in the architectural plan and metaspacescorresponding to real shelving structures in the store are referred toas “virtual shelving structures.”)

7.2 Planogram and Elevation Maps

The system can similarly access a preexisting planogram that visually,graphically, and/or textually defines relative positions of shelvingsegments and shelves, defines arrangements of products on these shelves,and contains addresses of these shelves, shelving segments, and shelvingstructures, etc. within shelving structures throughout the store. In oneimplementation, the system selects a set of elevation maps—from thepredefined planogram for the store—current as of the time the roboticsystem generated the real map of the store. Alternatively, the systemcan access these data from a product position database, such as in theform of a textual spreadsheet. However, the system can collect orretrieve such product location and assignment data for the store in anyother format.

7.3 Virtual Shelving Structures in the Architectural Metaspace

The system can then initiate the architectural metaspace by transferringvirtual shelving structures, virtual shelving segments, virtual aisles,virtual walls, virtual doors, and/or other virtual features from thevectorized architectural plan into a new vectorized line map defining anarchitectural metaspace for the store. The system can then linkpreexisting elevation maps—from the planograms—to corresponding virtualshelving structures in the architectural metaspace.

In one implementation, the system detects features in the vectorizedarchitectural plan that represent addresses or other identifiers ofshelving structures in the store. For example, the system can implementoptical character recognition to read a shelving structure addressdirectly from text in the architectural plan and can then label virtualshelving structures in the architectural metaspace with these shelvingstructure addresses based on proximity of a string of recognized text toa virtual shelving structure in the architectural plan. (The system cansimilarly recognize addresses of shelving segments, aisles, walls,doors, etc. in the architectural plan and write these addresses tocorresponding features in the architecture metaspace.) For each virtualshelving structure in the architectural metaspace, the system can matcha shelving structure address from a virtual shelving structure to ashelving structure address associated with a planogram—in the set ofelevation maps for the store—and then write a planogram ID to thevirtual shelving structure in the architectural metaspace. The systemcan thus link a planogram to a virtual shelving structure in thearchitectural metaspace based on like shelving structure address tags.

In another implementation, the planogram can also include a planogramlayout linking elevation maps of shelving structures throughout thestore to relative locations of these shelving structures within thestore. The system can thus distort (e.g., rotate, translate, globallyskew, locally skew, or otherwise manipulate) the planogram layout toalign locations of these elevation maps to virtual shelving structuresidentified in the architectural metaspace; and then transfers elevationmap identifiers from the planogram layout onto coinciding virtualshelving structures in the architectural metaspace.

In the foregoing implementations, the system can also identify virtualaisles between virtual shelving structures within the architecturalmetaspace and determine an orientation of a planogram—linked to avirtual shelving structure in the architectural metaspace—that faces theplanogram toward the adjacent aisle. However, the system can implementany other method or technique to link an existing planogram for thestore to a virtual shelving structure in the architectural metaspace

7.4 Virtual Shelving Segments in the Architectural Metaspace

For a virtual shelving structure represented in the architecturalmetaspace, the system can delineate shelving segments along the virtualshelving structure by mapping absolute or relative positions of shelvingsegments defined in a planogram matched to the virtual shelvingstructure, as described above. In one example, the system: extractsabsolute or relative dimensions of shelving segments and addresses ofshelving segments from a planogram; maps these shelving segmentdimensions to a corresponding virtual shelving structure in themetaspace to define positions of virtual shelving segments along thevirtual shelving structure; and then label each shelving segment in thevirtual shelving structure in the architectural metaspace with acorresponding address from the planogram. In another example, the systemcan: transform a 2D planogram into a 1D set of lines representingrelative positions of shelving segments within a shelving structure andlabel these lines with corresponding shelving segment addresses; anchorone end of the 1D line set to an end of the virtual shelving structurein the metaspace and skew the 1D line set to terminate on an oppositeend of the virtual shelving structure; define virtual shelving segmentswithin the virtual shelving structure in the architectural metaspaceaccording to these overlaid lines in the 1D line set; and write theshelving segments addresses from these overlaid lines in the 1D line setto corresponding virtual shelving segments thus defined for the virtualshelving structure in the architectural metaspace. In yet anotherexample, the system can: retrieve a standard shelving segment width(e.g., 4 feet) for the store; delineate virtual shelving segments withinthe virtual shelving structure based on this standard shelving segmentwidth; and transfer shelving segment addresses from nearest shelvingsegments defined in an existing planogram—matched to the virtualshelving structure—to these virtual shelving segments within the virtualshelving structure.

The system can then repeat any one or more of these processes for eachvirtual shelving structure in the architectural metaspace. However, thesystem can implement any other methods or techniques to link an existingplanogram to a virtual shelving structure in the metaspace, to defineshelving segments in the virtual shelving structure from shelvingsegment dimensions and positions defined in an existing planogram, andto label subregions of virtual shelving segments in the architecturalmetaspace with shelving segment addresses from an existing planogram (orproduct position database, etc.).

The system can also implement similar methods and techniques to labelsubregions of virtual shelving segments in the metaspace with shelfaddresses and vertical locations of shelves extracted from the set ofelevation maps. For example, the system can label each shelving segmentin the metaspace with a total height, a number of shelves contained inthe shelving segment, and/or vertical positions of shelves in theshelving segment, etc. The system can also label the architecturalmetaspace with category addresses, aisle addresses, and/or slotaddresses extracted from the set of elevation maps.

The system can therefore aggregate data from an architectural plan (orstandardized store layout) and preexisting planograms from the store toform an architectural metaspace for the store, such as including avectorized 2D line map indicating anticipated positions and addresses ofshelving structures and shelving segments within the store. Thearchitectural metaspace can thus define a “planned” representation ofthe store, such as including planned locations and dimensions of aisles,shelving structures, shelving segments, etc. within the store, etc.

7.5 Manual Shelving Structure Identification

The system can additionally or alternatively interface with a user(e.g., an employee of the store or a robotic system operator) toidentify and label shelving structures in the architectural plan. In oneexample, the system interfaces with the operator portal to present the(vectorized) architectural plan to the user. In this example, the usercan indicate regions within the architectural plan that correspond toshelving structures, such as by dragging a cursor over shelvingstructures shown in the architectural plan. Thus, as the user draws acontiguous cursor input over a region in the architectural plan, thesystem can select a set of lines within the architectural plan thatcompletely bounds this contiguous cursor input and then label this setof lines at a shelving structure. In this example, the user can also tagor label shelving structures in the architectural plan, such as byentering textual shelving structure addresses over identified shelvingstructures in the architectural plan. Furthermore, the system caninterface with the operator portal to collect user inputs identifyingand labeling categories, aisles, shelving segments, and/or shelves, etc.within the architectural plan.

The system can implement similar methods and techniques to prompt a user(e.g., an operator or manager of the store) to manually provide shelvingstructure addresses for a subset of shelving structures represented inthe architectural plan, such as for shelving structures—represented inthe architectural plan—for which low confidence in accuracy of extractedaddresses was calculated.

The system can thus access or initiate the architectural metaspace thatincludes a set of lines corresponding to user-confirmed shelvingstructures and user-supplied shelving structure addresses. As describedabove, the system can then transfer data from a preexisting planogram ofthe store into the architectural metaspace, such as shelving segmentlocations and addresses and shelf locations and addresses, based on likeshelving structure addresses tagged to virtual shelving structures inthe architectural metaspace and to the planogram of the store. However,the system can implement any other method or technique to identifyaisles, shelving structures, shelving segments, and/or shelves, etc.within the architectural metaspace and to label these virtual elementswith corresponding addresses, IDs, or other pointers to data stored inthe planogram or a product position database for the store.

7.6 Preexisting Architectural Metaspace

Alternatively, the system can access a preexisting architecturalmetaspace, such as uploaded by a user through an instance of theoperator portal executing on a desktop computer or other computingdevice. However, the system can implement any other method or techniqueto generate or access an architectural metaspace for the store.

8. Normalized Metaspace

Block S140 of the method S100 recites generating a normalized metaspaceby aligning features in the architectural metaspace to features in the(real) map. Generally, in Block S140, the system aligns virtual elementswithin the architectural metaspace to real features within the real mapof the store—collected from the robotic system in Block S120—therebytransforming the architectural metaspace into a normalized metaspacecontaining labeled elements representing real positions of aisles,shelving structures, and/or shelving segments, etc. within the store. Inparticular, the robotic system (or the computer system) calculates thereal map from scan data collected by the robotic system navigatingthrough the store, and the real map therefore represents a true layoutof the store, including real deviations from a unique architectural planor from a standardized layout for the store, such as: real gaps betweenshelving structures; misalignment of adjacent shelving structures;omitted or added shelving structures and shelving segments; positions ofreal structural elements (e.g., columns) within the store; etc.

The system can employ the real map as a master layout for the store andalign features in the architectural metaspace to features in this masterlayout to form the normalized metaspace such that waypoints generatedfrom the normalized metaspace in Block S160 are located according totrue positions of shelving structures, shelving segments, etc. in thestore rather than approximate positions of shelving structures, shelvingsegments, etc. as defined in an architectural plan or architecturalmetaspace. Therefore, in Block S140, the system can generate anormalized metaspace representing real gaps between shelving structures,misalignment of adjacent shelving structures, shelving structures in thestore omitted from the architectural plan, shelving structures in thestore added over the architectural plan, and positions of shelvingsegments within the store, etc.

In one implementation, the system scales and rotationally aligns thearchitectural metaspace to the real map by matching large or “global”features (clusters of shelving structures, doors, perimeter walls, etc.)in the architectural metaspace to like features in the real map. Thesystem then: associates virtual shelving structures identified in thearchitectural metaspace with features (e.g., lines, lines of adjacentpoints) in the real map based on positional and geometric similarities;and locally skews, scales, rotates, translates, or otherwise modifiesline vectors representing shelving structures in the architecturalmetaspace to locally align with associated features (e.g., lines, linesof adjacent points) in the real map, as shown in FIG. 1, therebytransforming the architectural metaspace into a normalized metaspacecontaining a vectorized line map representing real locations of shelvingstructures within the store and labeled with addresses of shelvingstructures, shelving segments, etc. linking such features to apreexisting planogram (or a product position database) for the store.

For example, the system can: detect shelving structures represented inthe map in Block S120, as described above; detect shelving structuresrepresented in the architectural metaspace in Block S130, as describedabove; and then calculate a transform that approximately centersshelving structures detected in the map onto shelving structuresdetected in the architectural metaspace in Block S140. By then applyingthe transform to shelving structures and to locations of shelvingstructure addresses in the architectural metaspace, the system cangenerate the normalized metaspace.

In the foregoing example, the system can: access a standardized storelayout—for a retail brand associated with the store—defining generictarget locations of shelving structures in stores associated with theretail brand; detect shelving structures represented in the standardizedstore layout in Block S130; and then calculate the transform thatapproximately centers a first subset of shelving structures detected inthe map to a first subset of shelving structures detected in thestandardized store layout (e.g., with greater than a thresholdconfidence that such features represent shelving structures) in BlockS140. The system can then initialize the normalized metaspace withlocations and addresses of the first subset of shelving structures byapplying the transform to the architectural metaspace. However, thesystem can also identify a second subset of shelving structures in themap that deviate from locations of nearest shelving structures in thestandardized store layout by more than a threshold distance or that arenot matched to shelving structures in the standardized store layout withmore than a threshold degree of confidence. The system can then prompt auser to manually link the second subset of shelving structures in themap to a second subset of shelving structures in the standardized storelayout, to provide addresses for these unknown shelving structures,and/or to mark some or all of these shelving structures as not needed ornot representative of shelving structures. Finally, the system can:insert the second subset of shelving structures (or remaining unknownshelving structures) in the map into the normalized metaspace; writeaddresses from the second subset of shelving structures in thearchitectural metaspace onto the second subset of shelving structures inthe normalized metaspace based on manual links provided by the user;and/or write addresses to the shelving structures as provided manuallyby the user.

The system can implement similar methods and techniques to: associatevirtual shelving segments identified and labeled in the architecturalmetaspace with features in the real map; and locally skew, scale,rotate, translate, or otherwise modify line vectors representingshelving segments in the architectural metaspace to locally align withassociated features in the real map in order to transform thearchitectural metaspace into a normalized metaspace.

The system can therefore relate labeled features in the architecturalmetaspace to features in the real map, locally reposition labeledfeatures in the architectural metaspace to align with related featuresin the real map to generate a normalized metaspace in Block S140, andtransfer shelving structure addresses (i.e., labels), shelving segmentaddresses, etc. from the architectural metaspace to the normalizedmetaspace in Block S140. However, the system can implement any othermethod or technique to transform the labeled architectural metaspaceinto a normalized metaspace that represents real positions of knownshelving structures (and/or other elements) within the store.

9. Store Coordinate System

Block S150 of the method S100 recites defining a coordinate system forthe store within the normalized metaspace. Generally, in Block S150, thesystem locates a coordinate system within the normalized metaspace inpreparation to calculate lateral distances, longitudinal distances, andorientations (e.g., yaw angle) for waypoints relative to a coordinatesystem in Block S160.

In one implementation, the system labels a location of the dock—placedin the store—in the normalized metaspace based on position data receivedfrom the robotic system in Block S120 and then defines an origin of thecoordinate system that coincides with the dock in the normalizedmetaspace. In this implementation, the system then aligns an x-axis ofthe coordinate system to a longest linear wall identified in thenormalized metaspace. However, the system can locate and position acoordinate system within the normalized metaspace in any other way oraccording to any other parameters.

10. Waypoint Generation

Block S160 of the method S100 recites defining a set of waypoints withinthe normalized metaspace based on positions of shelving structureswithin the normalized metaspace and imaging capabilities of the roboticsystem, wherein each waypoint defines a location and orientationrelative to the coordinate system; and Block S170 of the method S100recites labeling each waypoint in the set of waypoints with an addressof a shelving structure proximal the waypoint. Generally, in Block S160,the system generates a list of waypoint defining positions within thestore—relative to the coordinate system—at which the robotic system isto navigate and capture digital photographic images based on positionsof labeled shelving structures shown in the normalized metaspace, asshown in FIGS. 1 and 6. In Block S170, the system writes addresses ofshelving structures or shelving segments, etc. expected to be in thefield of view of a camera in the robotic system at each waypoint basedon addresses of nearby shelving structures or shelving segments, etc.shown in the normalized metaspace.

In one implementation, the system generates one waypoint for eachshelving segment labeled in the normalized metaspace. For example inBlock S160, the system can: access—from a planogram of the store—a firstelevation map associated with a first address of the first shelvingstructure, the first elevation map specifying locations of productfacings assigned to a first side of the first shelving structure;extract locations of a first set of shelving segments in the firstshelving structure from the first elevation map; and project locationsof the first set of shelving segments onto the location of the firstshelving structure in the normalized metaspace to locate shelvingsegments in the first shelving structure in the normalized metaspace. Inthis example, the system can then define waypoints longitudinallycentered between each shelving segment in the first shelving structurerepresented in the normalized metaspace; and calculate an angularorientation (e.g., yaw angle) normal to a longitudinal face of the firstshelving structure, relative to the coordinate system, in the normalizedmetaspace; and write the angular orientation to each waypoint in thefirst subset of waypoints. Therefore, for each waypoint, the systemcenters a waypoint in between a corresponding shelving segment in ashelving structure represented in the normalized metaspace such that theshelving segment is centered laterally in the field of view of a camerain the robotic system when the robotic system has navigated to alocation and orientation—relative to the coordinate system—defined bythe waypoint.

In this implementation, the robotic system can include a set of camerasarranged on one side of its mast; and the system can access a presettarget imaging distance for cameras in the robotic system and thendefine waypoints along a shelving structure represented in thenormalized metaspace such that each camera in the robotic system isoffset from the face of an adjacent shelving structure by this presettarget imaging distance when occupying locations and orientationsdefined by these waypoints.

Alternatively, the system can retrieve optical capabilities of camerasin the robotic system (e.g., zoom range, resolution), positions ofcameras in the robotic system, and image resolution requirements fortranslation into product stocking data, etc. and set a distance of eachwaypoint from its corresponding shelving segment based on these data andimage requirements.

For example, the system can: extract a height of a first shelvingstructure—represented in the normalized metaspace—from a first elevationmap associated with the address of the first shelving structure;calculate a target lateral offset distance from the first shelvingstructure based on the height of the first shelving structure andoptical capabilities of a camera (e.g., zoom, resolution) integratedinto the robotic system; and then define a first subset of waypointsoffset laterally from a longitudinal face of the first shelvingstructure by this target lateral offset distance. In another example,the system can locate a waypoint at a particular distance in front of acorresponding shelving segment that enables the two cameras offsetvertically along the robotic system's mast to capture the full height ofthe shelving segment, accounting for positional accuracy of the roboticsystem (e.g., +/−4″ from a target waypoint location, +/−1° from a targetyaw angle), while maximizing resolution of images captured by the twocameras. The system can thus write a position and orientation, such asin the form of a (x,y,∂) coordinate relative to the coordinate system,to a waypoint for each shelving segment labeled in the normalizedmetaspace.

In another implementation, for each virtual shelving segment labeled inthe normalized metaspace, the system can define a virtual area centeredin front of a shelving segment and for which the full height of theshelving segment will be within fields of view of cameras on one side ofthe robotic system when the robotic system is at any position within thearea (while accounting for positional accuracy of the robotic system,such as described above). For the robotic system that includes a set ofcameras on one side (e.g., the left side) of the mast and a second setof cameras on an opposing side (e.g., the right side) of the mast, asdescribed above, the system can then detect two overlapping virtualareas corresponding to opposing shelving segments within the normalizedmetaspace, calculate the intersection of these two overlapping virtualareas, and compress the two corresponding waypoints to a single waypointdefining a (x,y,∂) coordinate—with the intersection—at which the twoopposing shelving segments will be fully within the fields of view ofcameras on both sides of the robotic system, as shown in FIG. 1. Thesystem can thus compress two waypoints corresponding to two opposingshelving segments (i.e., two shelving segments facing each other acrossa common aisle) into a single waypoint centered laterally adlongitudinally between the two shelving segments. Therefore, for twoopposing shelving structures separated by an aisle of width not greaterthan a maximum imaging distance for the height of the shelvingstructure, the system can generate a row of waypoints along the aisle,including one waypoint centered across the aisle and centered betweenends of a shelving segment for each pair of opposing shelving segmentsalong the aisle.

However, for shelving structures that are too tall and/or for aislesthat are too wide to permit capture of images of shelving segments onboth sides of the robotic system at one waypoint, the system cangenerate two rows of waypoints along the aisle, wherein each row ofwaypoints defines positions along the aisle at which the robotic systemcaptures images of shelving segments within one adjacent shelvingstructure.

The system can thus generate a set of waypoints that define locationsand orientations (e.g., a set of (x,y,∂) coordinates in the coordinatesystem for the store) for which each shelving segment in the store isfully in the field of view of at least one camera in the robotic systemat least once when the robotic system navigates to each waypoint in thestore during a imaging routine. Alternatively, the system can generate aset of waypoints for a subset of shelving segments within the store. Forexample, the system can label each virtual shelving structure or eachvirtual shelving segment in the normalized namespace with a category(e.g., cosmetics, boxed cereal, canned goods, detergent, loose produce,loose deli meats) and/or shelving type (e.g., pegboard, flat shelf,glass deli case, open basket for loose produce, etc.) stored in acorresponding planogram or product position database. In this example,the system can generate waypoints for shelving segments corresponding tocosmetics, boxed cereal, canned goods, and detergent categories and/orfor shelving segments with pegboard-type and flat shelf-type labels, butthe system can exclude waypoints for shelving segments corresponding toloose produce and loose deli meats and/or for shelving segments labeledfor glass deli cases and open baskets for loose produce. However, thesystem can selectively generate waypoints for a subset of shelvingsegments (or a subset of shelving structures, shelves, etc.) within thestore based on any other shelving segment category, type, or otherparameter.

For each waypoint, the system can also write—to a waypoint—addresses ofcameras in the robotic system that are to record images once the roboticsystem has navigated to a waypoint (e.g., navigated to within athreshold distance and angular offset from the (x,y,∂) coordinatedefined by the waypoint), as shown in FIG. 1. For example, for a firstwaypoint at which the right side of the robotic system faces a shelvingsegment four feet in height but the left side of the robotic system doesnot face a nearby shelving segment, the robotic system can write anaddress for a lower-right camera in the robotic system to the firstwaypoint in order to trigger the lower-right camera to capture an imageof the short shelving segment when the robotic system reaches the firstwaypoint. Similarly, for a second waypoint at which the right side ofthe robotic system faces a shelving segment seven feet in height but theleft side of the robotic system does not face a nearby shelving segment,the robotic system can write an address for a lower-right camera and anaddress for an upper right camera in the robotic system to the secondwaypoint in order to trigger the lower-right camera and the upper-rightcamera in the robotic system to capture images of the tall shelvingsegment when the robotic system reaches the second waypoint.Furthermore, for a third waypoint at which the right side of the roboticsystem faces a first shelving segment seven feet in height and a leftside of the robotic system faces a second shelving segment seven feet inheight, the robotic system can write addresses for a lower-right camera,an upper-right camera, a lower-left camera, and an upper-left camera inthe robotic system to the third waypoint in order to trigger the allfour cameras in the robotic system to capture images of the two tallshelving segments when the robotic system reaches the third waypoint.However, the system can label a waypoint with an address of a cameraaccording to any other parameter or schema.

The system can also associate each camera address in a waypoint with anaddresses of a shelving structure, a shelving segment, a shelf, and/or aslot, etc. expected to be in a field of view of the corresponding camerawhen the robotic system is proximal the location and orientation—withinthe store—defined in the waypoint, as shown in FIG. 1. In one example,based on a physical position of a camera—addressed in a waypoint—on theremote controller, the location defined in the waypoint, the orientationdefined in the waypoint, and labeled features contained in thenormalized metaspace, the system can predict a shelving structure, ashelving segment, a shelf, and/or a slot, etc. that will be in the fieldof view of a camera once the robotic system has navigated to thewaypoint. In this example, the system can then collect addresses of thisshelving structure, shelving segment, shelf, and/or slot, etc. predictedto fall in the field of view of the camera from normalized metaspace andassociate these addresses with the corresponding camera address in thewaypoint. Thus, when the robotic system navigates to a waypoint andcaptures an image through a particular camera as specified in thewaypoint, the robotic system can write the address of a shelvingstructure, shelving segment, shelf, and/or slot, etc.—associated withthe address of the particular camera in the waypoint—to the imagemetadata, and the system can automatically match the image to aplanogram, predict SKUs of products shown in the image, and selectlabeled template images to match to regions in the image based on theseshelving structure, shelving segment, shelf, and/or slot addresses.

In the foregoing example, for each waypoint in the set of waypoints, thesystem can label the waypoint with an address of a particular shelvingstructure and a particular shelving segment predicted to fall within afield of view of the optical sensor when the robotic system occupies alocation and an orientation defined by the waypoint in Block S160.During a subsequent imaging routine, the system can: receive a firstimage recorded by the robotic system while occupying a first waypoint,in the set of waypoints, adjacent a first shelving segment in the firstshelving structure; access a first list of identifiers of productsassigned to the first shelving segment based on a first address of thefirst shelving segment stored with the first waypoint; retrieve a firstset of template images, from a database of template images, representingproducts in the first list of products stored with the first waypoint;and identifying products on shelves in the first shelving segment basedon similarities between features detected in the first image andtemplate images in the first set of template images, as described below,in Block S190.

The system can therefore generate a set of waypoints specific to motionand imaging specifications of a robotic system placed in the store, suchas specific to positional precision locating capabilities of the roboticsystem, positions of cameras in the robotic system, zoom and resolutioncapabilities of cameras in the robotic system, etc. However, the systemcan implement any other method or technique: to define a location andorientation of one or more waypoints for the store; to specify one ormore cameras on the robotic system to capture images at a waypoint; orto determine a shelving structure, a shelving segment, a shelf, and/or aslot, etc. within the field of view of a camera at a waypoint and toassociate a callout for image capture by the camera with an address ofthis shelving structure, a shelving segment, a shelf, and/or a slot,etc.

11. Path Planning

As shown in FIG. 5, one variation of the method S100 includes BlockS162, which recites: defining an order for the set of waypoints based onproduct values of products assigned to slots, within the set of shelvingstructures, in fields of view of cameras integrated into the roboticsystem when occupying each waypoint in the set of waypoints; and servingthe order for the set of waypoints to the robotic system for executionduring the imaging routine. Generally, in Block S162, the system candefine an order for waypoints defined in Block S160.

For example, the system can: calculate a linear combination of values ofproducts assigned to a shelving segment by the planogram for eachshelving segment in each shelving structure represented in thenormalized metaspace; assign a value to a waypoint based on a linearcombination of values of products assigned to a shelving segmentadjacent the waypoint for each waypoint in the set of waypoints; anddefine an order for the set of waypoints based on values assigned to theset of waypoints in Block S162. In this example, the system cancalculate the order for the set of waypoints: that minimizes totaldistance traversed by the robotic system from a docking station withinthe store, to a first waypoint adjacent a first shelving segmentassociated with a greatest linear combination of values of assignedproducts, to a second waypoint adjacent a second shelving segmentassociated with a lowest linear combination of values of assignedproducts, and back to the docking location; minimizes energy consumptionby the robotic system during a next imaging routine; or achieves anyother one or more weighted parameters, as described below. The systemcan then serve the set of waypoints and the order for the set ofwaypoints to the robotic system for execution during the imaging routinein Block S180.

11.1 Individual Waypoint Ranking

In this variation, the system can calculate a value of each waypoint inthe store, rank these waypoints by value, and set an order fornavigating through these waypoints accordingly, as shown in FIG. 5.

In one implementation, the system accesses a planogram and a map of thestore (e.g., a map geo-referenced to the planogram) to calculate a totalproduct value (described below) of products in the field of view ofcameras integrated into the robotic system at each waypoint in the setof waypoints. For example, for a particular waypoint, the system can:identify the location of the particular waypoint within the planogram;project known fields of view of cameras integrated into the roboticsystem onto the planogram—based on an orientation of the robotic systemspecified in the waypoint and based on a distance from the waypoint toan adjacent shelving structure defined in the map of the store—toidentify a set of slots (e.g., a set of slot addresses) on the shelvingstructure that may be captured in a set of images recorded by therobotic system occupying the waypoint; pass this set of slot addressesinto the planogram to retrieve a set of SKUs assigned to these slots;and then associate this waypoint with one or more unique SKUs and anumber of facings of each SKU in the field of view of the robotic systemwhen occupying the particular waypoint in real space in an orientationdefined by the waypoint. In this example, the system can repeat thisprocess for each waypoint defined throughout the store in order toassociate each waypoint with a unique set of SKUs and SKU facings. Thesystem can store and implement these SKUs and SKU facings over a seriesof imaging routines; the system can also repeat this process each timethe planogram is revised or each time a new planogram is activated forthe store.

The system can then rank the set of waypoints by product values of SKUsassociated with each waypoint. For example, for each SKU assigned to aslot in the current planogram, the system can calculate a “productvalue” (or “priority,” or “weight”) as a function of profitability (or“margin”) associated with the SKU and the number of units of the SKUsold at the store per unit of time (e.g., per day). In this example, thesystem can calculate a product value of a particular SKU as a product ofthe margin and a sale rate of the SKU, that is:

[product value]=[margin]·[# units sold per day].

The system can then rank waypoints by greatest product value of a singleSKU assigned to a slot in the field of view of the robotic system whenoccupying each waypoint. Alternatively, for each waypoint, the systemcan calculate an “aggregate product value” for a particular waypoint bysumming product values of SKUs assigned to all discernible slots in thefield of view of the robotic system when occupying the particularwaypoint; the system can then rank waypoints by greatest aggregateproduct value.

In another implementation, the system can rank waypoints based on nearreal-time POS sales data. In this implementation, in preparation for(e.g., immediately before) dispatching the robotic system to execute aimaging routine in Block S110, the system can access sales data from aPOS system installed in the store to identify a quantity of each SKUsold at the store over a preceding period of time, such as: within afive-minute, thirty-minute, one-hour, or eight-hour period leading up tothe current time; since a last restocking period at the store; or sincea last imaging routine completed by the robotic system. The system canthen calculate a current product value for each SKU in the store basedon a margin assigned to each SKU (e.g., written to remote memory by aregional manager through an administrator portal hosted by the system)and a real-time frequency of sales of the SKU, such as:

[product value]=[margin]·[# units sold since last imaging routine].

For each waypoint defined throughout the store, the system can thusrecalculate an aggregate product value for a particular waypoint basedon new product values representing recent sale frequencies of SKUsfalling within the field of view of the robotic system when occupyingthe particular waypoint. The system can then rank or prioritize eachwaypoint accordingly.

In a similar implementation, the system can prioritize waypoints basedon historical sales frequencies. For example, in preparation fordispatching the robotic system to execute a imaging routine in BlockS110, the system can retrieve historical sale data for SKUs specified ina current planogram of the store, such as: sale data for a last day,week, month, quarter, or year; for a current day of the week, sale datafrom this same day of the week for the previous five weeks; for acurrent or upcoming holiday, sale data from the last instance of thesame holiday or holiday season; for a rainy spring day, sale data fromthe last two instances of rainy spring days; etc. In this example, thesystem can: retrieve these sale data from a POS implemented by thestore; or implement methods and techniques described below to extractthese sale data from changes in numbers of product facings identifiedacross images collected during past imaging routines. For each SKU, thesystem can then multiply a stored margin of a particular SKU by thehistorical number or frequency of sales of the particular SKU tocalculate a new product value for the SKU; for each waypoint, the systemcan calculate a new aggregate product value, as described above. Thesystem can then automatically re-rank waypoints throughout the storeaccordingly by aggregate product value.

11.2 Waypoint Ordering by Individual Waypoint Value

The system can then define an order in which the robotic system is tonavigate through the set of waypoints during the upcoming imagingroutine based on waypoint rankings.

In one implementation, the system orders the set of waypoints such that,when executed by the robotic system, the robotic system: navigates froma home location at a dock to a first waypoint associated with a greatestaggregate product value in the set; executes an imaging routine tocapture a set of images at each other waypoint between the home locationand the first waypoint; marks each of these other waypoints ascompleted; upon reaching the first waypoint, executes an imaging routineto capture a set of images at the first waypoint; and then marks thefirst waypoint as completed. In this implementation, the system ordersthe remaining set of waypoints such that, after collecting images at thefirst waypoint, the robotic system: navigates toward a secondwaypoint—associated with a greatest aggregate product value for theremaining set of waypoints—via a path that minimizes overlap withwaypoints already completed; executes an imaging routine to capture aset of images at each other waypoint between first waypoint and thesecond waypoint; marks each of these other waypoints as completed; uponreaching the second waypoint, executes an imaging routine to capture aset of images at the second waypoint; and marks the second waypoint ascompleted; etc. until a set of images has been recorded at all waypointsdefined for the store or until the imaging routine is terminated, asdescribed below.

In another implementation, the system calculates an order for navigatingthrough the set of waypoints that maximizes total product value ofproducts imaged by the robotic system per distance traversed by therobotic system during a imaging routine.

In another implementation, the system: calculates a sum of productvalues of products assigned to slots within a field of view of therobotic system occupying a waypoint for each waypoint in the store;identifies a first subset of waypoints associated with sums of productvalues greater than a threshold product value; identifies a secondsubset of waypoints associated with sums of product values less than thethreshold product value; and defines an order for the set of waypointsthat specifies navigation to (and image capture at) each waypoint in thefirst subset of waypoints, followed by navigation to each waypoint inthe second subset of waypoints, followed by navigation to each waypointin the first subset of waypoints during the imaging routine. In thisimplementation, the system can separately order the first and secondsubsets of waypoints to achieve a shortest total distance traversed overthe course of the imaging routine in light of the robotic system's startlocation within the store. Alternatively, the system can: calculate anorder for the complete set of waypoints that minimizes overlap along therobotic system's trajectory; order the first subset of waypoints byaggregate product value at each waypoint, to maximize the product valueof product images per unit time or per unit distance traversed by therobotic system, or to minimize overlap in the robotic system'strajectory, etc.; and then append the waypoint order for the completeset of waypoints with the ordered first subset of waypoints. Forexample, for a store defining an approximately rectangular floor planwith parallel rows of shelving structures, the system can: define aserpentine path up and down each aisle in sequence from a near end ofthe store to a far end of the store followed by (or preceded by) a looparound the perimeter of the store to capture images at all waypointswith minimum path overlap; calculate an order for the first subset ofwaypoints—associated with aggregate product values exceeding a thresholdvalue—that minimizes a path traversed by the robotic system on its wayback to its dock; and append the serpentine path with the ordered firstsubset of waypoints.

In Block S162, the system can thus select a subset of waypointsassociated with aggregate product values exceeding a threshold value (orselect a preset number of waypoints associated with the greatest productvalues in the set of waypoints) and define a trajectory for the roboticsystem that directs the robotic system to this subset of waypoints (atleast) twice and to all other waypoints in the store (at least) onceduring a imaging routine.

11.3 Waypoint Order by Shelving Structure Priority

Alternatively, in this variation, the system can group waypoints andimplement methods and techniques described above to set an order fornavigating between groups of waypoints within the store during a imagingroutine.

In one implementation, the system groups waypoints by row defined alongthe length of each shelving structure within the store. The system then:sums product values of products assigned to slots in a shelf segment foreach shelving structure in the store (i.e., for each group ofwaypoints); prioritizes the set of shelving structures in order ofproduct value sum; orders the groups of waypoints accordingly; andcalculates a trajectory through the ordered groups of waypoints thatminimizes the total distance traversed by the robotic system fromwaypoints defined along a first shelving structure associated with agreatest product value sum to waypoints defined along a last shelvingstructure associated with a lowest product value sum. Thus, when therobotic system is dispatched to execute a imaging routine in Block S110,the robotic system can: navigate to the first shelving structure andcapture images at each waypoint in the first group of waypoints; thennavigate to the second shelving structure and capture images at eachwaypoint in the second set of waypoints, even if the second shelvingstructure is substantially removed from the first shelving structure;etc. for each shelving structure in the store. In a similarimplementation, the system can rank each shelving structure (e.g., eachgroup of waypoints) by the greatest product value of a single SKUassigned to each shelving structure and order the groups of waypointsaccordingly.

In another implementation, the system can calculate an order fornavigating through groups of waypoints that maximizes the total productvalue of products imaged per unit time throughout a imaging routine. Inone example: the sum of all product values of products assigned to slotsalong shelving structure 2 within the store is the greatest at 4389; thesum of all product values of products assigned to slots along shelvingstructure 1—facing shelving structure 2—is 1238; the sum of all productvalues of products assigned to shelving structure 18—near an oppositeend of the store—is second greatest at 4285; the sum of all productvalues of products assigned to shelving structure 9—near an opposite endof the store—is 2831; and the robotic system is stationed nearestshelving structure 18. In this example, the system can set an order fornavigating through waypoints in the store such that the robotic systemfirst images shelving structure 18, . . . followed by shelving structure9, . . . followed by shelving structure 2, and finally shelvingstructure 1, thereby maximizing the product value of products imaged bythe robotic system per unit time throughout the imaging routine.

In the foregoing implementations, the system can implement methods andtechniques described above to (re)calculate product values for SKUsarranged throughout the store. The system can also: identify selectshelving structures containing singular high-value products (e.g.,products associated with static or dynamic product values exceeding athreshold product value) or associated with total product valuesexceeding a threshold total product value; and define a trajectory forthe robotic system that directs the robotic system along these selectshelving structures (at least) twice and along all other shelvingstructures (at least) once during a imaging routine.

11.4 Waypoint Order by Scheduled Restocking Periods

In another implementation, the system orders waypoints based onscheduled restocking periods. For example, the system can: access arestocking schedule for each section (or “category”) of the store (e.g.,a spice section, a canned foods section, a laundry section, a carbonatedbeverage section, and a dairy section, etc.); identify a particularsection of the store assigned an upcoming restocking period; retrieve agroup of waypoints within this particular section; and generate awaypoint order that places the group of waypoints ahead of all otherwaypoints in the store for a next scan period. In this example, thesystem can also implement methods and techniques described above toorder the group of waypoints corresponding to the particular section ofthe store and the remaining set of waypoints in the store, such as byordering the group of waypoints and the remaining set of waypoints tominimize a total distance traversed by the robotic system over theimaging routine.

Alternatively, in preparation for a scheduled restocking period for theparticular section of the store, the system can append an ordered listof waypoints—generated as described above—with a group of waypointscorresponding to the particular section of the store. During the imagingroutine, the system can generate and serve real-time prompts to restockhigh-value slots in Block S140, as described below, based on image datacollected during the imaging routine. Upon conclusion of the imagingroutine, the system can generate a restocking list specific to theparticular section of the store in Block S150 based on images recordedat the end of the imaging routine such that the section-specificrestocking list is most current upon commencement of the restockingperiod for the particular section of the store.

The system can implement similar methods and techniques in response toreceipt of manual confirmation of a restocking period for a particularsection of the store, such as received through a manager portal, asdescribed below.

11.5 Waypoint Order by Scheduled Delivery

In another implementation, the system can access a delivery schedule forthe store and order waypoints for an upcoming imaging routine based onproduct scheduled for upcoming delivery. For example, the system canaccess a delivery schedule indicating that a carbonated beveragedelivery vehicle is scheduled to make a delivery of carbonated beverageswithin one to two hours. The system can thus trigger the robotic systemto execute a imaging routine in Block S110. In Block S162, the systemcan also serve an ordered set of waypoints to the robotic system,including: a first group of waypoints defined along a carbonatedbeverage aisle in the store; followed by a second group of discretewaypoints adjacent discrete carbonated beverage displays (e.g.,refrigerators) arranged throughout the store; followed by waypoints forthe remainder of the store. Upon conclusion of the first and secondgroups of waypoints, the system can generate a carbonated beveragerestocking list for the carbonated beverage and discrete carbonatedbeverage displays throughout the store in Block S150 based on image datacollected from the robotic system during the imaging routine thusexecuted by the robotic system; associates of the store (or a carbonatedbeverage representative delivering product to the store) can then accessthis carbonated beverage restocking list to restock these slotsthroughout the store immediately upon arrival of the delivery vehicle.

Alternatively, in preparation for a scheduled delivery of carbonatedbeverages, the system can append an ordered list of waypoints—generatedas described above—with the first group of waypoints defined along acarbonated beverage aisle in the store followed by the second group ofdiscrete waypoints adjacent discrete carbonated beverage displaysarranged throughout the store. During the imaging routine, the systemcan generate and serve real-time prompts to restock high-value slots inBlock S140, as described below, based on image data collected during theimaging routine. Upon conclusion of the imaging routine, the system cangenerate a carbonated beverage restocking list specific to slotsassigned carbonated beverages in Block S150 based on images recorded atthe end of the imaging routine such that the carbonated beveragerestocking list is most current upon arrival of the carbonated beveragedelivery vehicle.

11.6 Waypoint Order by Manual Selection

In yet another implementation, the system can: receive a selection of aregion of interest within the store from a store manager, regionalmanager, or other associate of the store, such as through a managerportal; and then order waypoints for an upcoming imaging routine withwaypoints in this region of interest preceding all other waypoints inthe ordered set of waypoints for the store. The system can thenimplement methods and techniques described above to order these discretegroups of waypoints.

The system can thus store a set of waypoints for a store, wherein eachwaypoint designates capture of images of one or more shelves within thestore, and define an order for the robotic system to navigate throughthis set of waypoints during a imaging routine. In preparation for anupcoming imaging routine, the system can then push definitions for thesewaypoints and an order for these waypoints to the robotic system, suchas over a cellular network or through a computer network (e.g., theInternet), in Block S162. However, the system can maintain, update,and/or distribute waypoints including any other data or prompts in anyother format to one or more robotic systems placed within a store at anyother type, time, or frequency.

12. Imaging Routine

Block S180 of the method S100 recites uploading the set of waypoints tothe robotic system for execution in a subsequent imaging routine.Generally, in Block S180, the system serves the set of waypointsgenerated in Block S160 (and the waypoint order defined in Block S162)to the robotic system. For example, the system transmits the set ofwaypoints to the robotic system over the Internet via a local wirelessrouter installed in the store.

Along with the set of waypoints, the system can also upload a master mapof the store to the robotic system, such as in the form of the real mapgenerated from distance scans collected by the robotic system during anautonomous mapping routine, as described above, and labeled with thecoordinate system for the store. During a subsequent imaging routine,the robotic system can sequentially navigate through the set ofwaypoints by: scanning its local environment with its integrated mappingsensor (e.g., a LIDAR sensor); compiling scans from the mapping sensorinto a new map of the robotic system's environment; and determining itslocation within the store's coordinate system by comparing the new mapto the master map—defining the coordinate system—of the store. Therobotic system can thus navigate to a next waypoint by moving to aposition and orientation within the store at which the output of themapping sensor aligns—within a threshold distance and angle—with aregion of the master map corresponding to the (x,y,∂) location andorientation defined in the waypoint. The robotic system can capture andtag images at the waypoint, as described above, before navigating to anext waypoint defined for the store. During or upon completion of theimaging routine, the robotic system can upload images to the system forprocessing, as described above.

During or upon completion of the imaging routine, the robotic system canalso upload a new real map of the store (or raw scan data collected bythe mapping sensor during the imaging routine) to the system. The systemcan then repeat Blocks of the method S100 to recalculate a set ofwaypoints for the store and to load this new set of waypoints onto therobotic system. Therefore, by selecting a latest architectural plan,standardized store layout, and/or set of elevation maps for the storefollowing each imaging routine, the system can generate a currentarchitectural metaspace with current shelving structure, shelvingsegment, shelf, and/or slot addresses for the store. Furthermore, byaligning the current architectural metaspace to the newest real map ofthe store, the system can compensate for any latest physical changes tothe store's layout and generate a new set of waypoints for the storeaccordingly. However, the system can implement any other method ortechnique to update a set of waypoints for the store based on new datacollected by the robotic system and to supply these waypoints and newreal maps to the robotic system.

13. Product Identification and Restocking Prompts

As shown in FIG. 4, the system can then: receive images (e.g., digitalcolor photographic images) recorded by the robotic system during theimaging routine, such as in real-time, intermittently, or uponconclusion of the imaging routine; process these images to determine thestocking condition of the store; and issue notifications and/or arestocking list responsive to differences between stocking condition ofthe store and the planogram of the store in Block S190.

In one implementation, the system receives a first image recorded by afirst camera in the robotic system while occupying a firstwaypoint—adjacent a first shelving segment in the first shelvingstructure—during the imaging routine. The system then implements methodsand techniques described in U.S. patent application Ser. No. 15/600,527to: detect a first shelf in the first image; determine an address of thefirst shelf based on a first address of a first shelving structureassociated with the first waypoint and a vertical position of the firstshelf within the first image; retrieve a first list of products assignedto the first shelf by the planogram based on the address of the firstshelf; retrieve a first set of template images from a database oftemplate images, wherein each template image in the first set oftemplate images includes visual features of a product specified in thefirst list of products; extract a first set of features from a firstregion of the first image adjacent the first shelf; confirm presence ofa unit of a first product—in the first list of products—on the firstshelf in response to correlation between features in the first set offeatures and features in a first template image, in the first set oftemplate images, representing the first product; determine that a unitof the first product is improperly stocked on the first shelf inresponse to deviation between features in the first set of features andfeatures in the first template image; and generate a first restockingprompt for the first product on the first shelf in response todetermining that the unit of the first product is improperly stocked onthe first shelf in Block S190.

In the foregoing example, the system can also: project heights of afirst set of shelves assigned to the first shelving segment by the firstelevation map of the first shelving structure onto the first image basedon a known position of the first camera on the robotic system andidentify a first region in the image corresponding to a first shelf inthe first shelving segment accordingly; or implement template matching,object recognition, or other methods or techniques to identify the firstshelf in the first region of the first image. The system can thenextract a first list of identifiers of products assigned to the firstshelf by the first elevation map; retrieve a first set of templateimages, from a database of template images, representing products in thefirst list of products; and identify products on the first shelf basedon similarities between features detected in the first region of thefirst image and template images in the first set of template images inBlock S190.

The system can repeat this process for images of other shelvingstructures, shelving segments, shelves, etc. recorded by the roboticsystem while occupying other waypoints throughout the store during theimaging routine. The system can then aggregate presence, quantity,and/or locations or products detected in these images into a currentstocking condition of the store (e.g., a “realogram”) and provide thesedata to a manager or other associate of the store, such as through amanager or associate portal within a web browser or native applicationexecuting on a desktop computer or mobile computing device (e.g., asmartphone, a tablet). The system can also automatically compare thecurrent stocking condition of the store to the planogram to detectdifferences, aggregate these differences into a list of restockingprompts (e.g., ordered according to values of products specified inthese restocking prompts, and then serve these restocking prompts toassociates of the store in order to guide manual restocking of thestore, such as in real-time, during off-peak patron hours, or duringstore closure hours, as shown in FIG. 5 and such as described in U.S.patent application Ser. No. 15/347,689.

In this variation, when processing images recorded by the roboticsystem, the system can also extract shelf, shelving segment, shelvingstructure, and/or aisle addresses from shelf tags arranged on shelves,aisle signs arranged on shelving structures, etc. shown in these images.The system can then confirm addresses assigned to shelving structures inthe normalized metaspace according to such addresses extracted fromimages recorded by the robotic system when occupying waypoints at knownlocations proximal these shelving structures within the store. Forexample, the system can: receive a first image recorded by the roboticsystem while occupying a first location during the mapping routine; scanthe first image for a first aisle address; project the first aisleaddress onto the map based on the first location of the robotic systemwhen the first image was recorded and a known position of thecamera—integrated into the robotic system—that recorded the first image;and then confirm alignment of the architectural metaspace to the map ofthe store based on alignment between the first aisle address andaddresses imported from the architectural metaspace into the normalizedmetaspace. However, the system can implement any other methods ortechniques to automatically confirm alignment and detect deviationbetween the architectural metaspace to the map of the store.

14. Normalized Metaspace Variations

In one variation, the robotic system and the system cooperate togenerate a normalized metaspace for the store without an existingarchitectural plan or standardized store layout for the store. In thisvariation, throughout an autonomous mapping routine, the robotic systemintermittently captures images of shelving structures within the store.For example, the robotic system can implement simultaneous localizationand mapping techniques to construct and update a map of the interior ofthe store and can simultaneously implement template matching, patternmatching, pattern recognition, and/or other techniques to identifyaisles and shelving structures within the map. In this example, uponidentification of an aisle, the robotic system can automaticallynavigate to a position proximal a lateral and longitudinal center of theaisle, reorient itself to align its integrated cameras to adjacentshelving structures, record an image at each camera, and then write alocation and orientation of the robotic system within the map and anaddress of a corresponding camera to image metadata for each image thusrecorded. The robotic system can repeat this process throughout theautonomous mapping routine to generate a map of the store and to collectimages of various shelving structures within the store, and the roboticsystem can then return this map and images to the system. Alternatively,the robotic system can upload the real map to the system upon conclusionof the autonomous mapping routine; and the system can remotely andasynchronously identify aisles and shelving structures in the real map,as described above, and define one temporary waypoint including a(x,y,∂) coordinate adjacent each identified shelving structure, and thendispatch the robotic system on a reconnaissance routine to collectimages at each temporary waypoint. However, the robotic system and thesystem can implement any other method or technique to collect a currentreal map of the store and images of current shelving structures in thestore.

The system can then transform the real map into a vectorized line map ofthe store, define a coordinate system within the vectorized line map,and identify aisles, shelving structures, and/or shelving segments, etc.in the vectorized line map, as described above. The system can alsolocate images captured by the robotic system during the autonomousmapping routine (or during the reconnaissance routine) within thevectorized line map.

The system can then identify SKUs of products in the images and populateshelving structures identified in the vectorized line map with SKUs ofproducts identified in images captured nearby. For example, the systemcan access a database of template images, wherein each template image inthe database is tagged with a SKU or other identifier or descriptor of aproduct shown in the template image. In this example, the system canimplement template matching techniques to match a template image—fromthe template image database—to a region in an image recorded during theautonomous mapping routine, and the system can then transfer a SKU fromthe matched template image to a shelving structure identified in thevectorized line map and proximal a location at which the image wascaptured.

In particular, the system can: identify a particular aisle occupied bythe robotic system when the image was recorded based on an (x,y)coordinate stored in the image metadata; identify a particular shelvingstructure—of two shelving structures lining the particular aisle—shownin the image based on the address of the camera that recorded the imageand the orientation (e.g., yaw) of the robotic system stored in theimage metadata; and write the SKU stored with the matched template imageto the particular shelving structure in the vectorized line map. Thesystem can repeat this process for other products shown in the image inorder to populate one shelving structure identified in the vectorizedline map with SKUs of multiple products, such as multiple instances ofthe same SKU or multiple different SKUs. For example, the system canmatch regions in the image to labeled template and transfer SKUs frommatched template images to the shelving structure shown in thevectorized line map until the first of three unique SKUs or a total ofsix SKUs are identified in the image. The system can duplicate thisprocess for each other image captured by the robotic system during theautonomous mapping routine in order to populate the vectorized line mapof the store with one or more SKUs per identified shelving structure.

The system can then retrieve a set of current planograms (or a currentproduct position database) for the store, as described above, and canlink a particular planogram to a particular shelving structure shown inthe vectorized line map based on like SKUs represented in the planogramand tagged to the particular shelving structure. In particular, for ashelving structure tagged with one or more SKUs in the vectorized linemap, the system can identify an elevation map—from the preexistingplanogram of the store—designating the same set of SKUs. Similarly, thesystem can identify a planogram designating a greatest number of SKUs(or a threshold proportion of SKUs) tagged to the shelving structureshown in the vectorized line map, such as to accommodate possiblemisplaced products on the shelving structure at the time of theautonomous mapping routine. The system can then write an address of thematched planogram and/or a shelving structure address extracted fromwithin the matched planogram to the virtual shelving structure in thevectorized line map.

The system can then implement methods and techniques described above todelineate shelving segments in the virtual shelving structure based onpositions of shelving segments shown in the planogram and to writeshelving segment, shelf, and/or slot addresses, etc.—stored in thematched planogram—to the shelving structure shown in the vectorized linemap. The system can also duplicate this process for other shelvingstructures identified in the vectorized line map and transform theresulting labeled line map into a normalized metaspace. However, thesystem can implement any other method or technique to generate anormalized metaspace.

15. Manual Waypoints

Furthermore, the system can append a set of waypoints automaticallycalculated for the store, as described above, with waypoints enteredmanually by a user or based on data entered manually by the user. Forexample, following installation of a temporary display within the store,an employee of the store can access the operator portal at a computer,tablet or smartphone, open the normalized metaspace, select a pointwithin the normalized metaspace corresponding to the current location ofthe temporary display, and enter height, width, and identification datafor the temporary display. In this example, the system can thencalculate a location and orientation for a new waypoint, define a cameraaddress for the waypoint, and write identification data for thetemporary display to the waypoint before uploading the new waypoint tothe robotic system. However, the system can implement any other methodor technique to generate a waypoint based on user-supplied information.

16. Real-Time Waypoints and Temporary Shelving Structures!!

As shown in FIG. 6, one variation of the system includes Block S164,which includes, at the robotic system, during the imaging routine:detecting a matrix barcode in a field of view of a second optical sensorin the robotic system; calculating a temporary waypoint based on aposition and an orientation of the matrix barcode in the field of viewof the second optical sensor, the temporary waypoint defining a firstlocation and a first orientation that aligns a field of view of theoptical sensor to a temporary shelving structure coupled to the matrixbarcode; recording a first image through the optical sensor whileoccupying the temporary waypoint. Block S164 also includes: determiningan identity of the temporary shelving structure based on the matrixbarcode; based on the identity of the temporary shelving structure,retrieving a first list of products assigned to the temporary shelvingby a planogram of the store; retrieving a first set of template imagesfrom a database of template images, each template image in the first setof template images including visual features of a product specified inthe first list of products; confirming presence of a unit of a firstproduct, in the first list of products, on the temporary shelvingstructure in response to correlation between features in the first imageand features in a first template image, in the first set of templateimages, representing the first product; determining that a unit of asecond product is improperly stocked on the temporary shelving structurein response to deviation between features in the first image andfeatures in a second template image, in the first set of templateimages, representing the second product; and, in response to determiningthat the unit of the second product is improperly stocked on thetemporary shelving structure, generating a first restocking prompt forthe second product on the temporary shelving structure.

Generally, in Block S164, the system defines a temporary waypointautomatically and in real-time when a temporary shelving structure isdetected by the robotic system during an imaging routine; the roboticsystem can then navigate to this temporary waypoint and record an image,which the system can then process to determine a stocking condition ofthe temporary shelving structure. In one implementation, the roboticsystem can include a 2D or 3D depth sensor and a 2D or 3D forward-facingcamera directed outwardly from its front. During an imaging routine, therobotic system can regularly scan a field ahead of the robotic systemthrough the depth sensor and camera, map surfaces nearby based on thesescan data, and localize itself (e.g., relative to the normalizedmetaspace or a past map of the floor) based on locations of surfacesnearby, thereby enabling the robotic system to navigate from waypoint towaypoint with a high locational and rotational accuracy. However, duringthe imaging cycle, the system can also scan images recorded through theforward-facing camera for matrix barcodes (e.g., quick-response code,one-dimensional barcodes, etc.) or other optical identifiers. When amatrix barcode is detected, the robotic system (or the remote computersystem) can scan a local or remote directory, such as a name mappingsystem, for an identifier of this matrix barcode; if the directoryreturns an identifier indicating that the matrix barcode is attached toor is otherwise associated with a temporary shelving structure—such as afreestanding cardboard popup configured to store units of one or moreproducts associated with a particular brand—the system can execute BlockS164 to define and execute a temporary waypoint adjacent the temporaryshelving structure.

In the foregoing implementation, the robotic system (or the remotecomputer system) determines that the matrix barcode detected in an imagerecorded by the forward-facing camera is associated with a temporaryshelving structure, the robotic system retrieves information related tothe temporary shelving structure and associated with the matrix barcode,such as a height of the temporary shelving structure and the position ofthe matrix barcode on the temporary shelving structure relative to afront face of the temporary shelving structure from a remote temporaryshelving structure database. The robotic system can then calculate alocation and an orientation of a temporary waypoint that, when occupiedby the robotic system, centers the front face of the temporary shelvingstructure in a field of view of a side-facing color camera on a side ofthe mast of the robotic system and at a preset target offset distancebetween the side-facing color camera and the front face of the temporaryshelving structure, such as based on: these data collected from thetemporary shelving structure database; the location of the roboticsystem within the store at the time the matrix barcode was detected; theposition, orientation, and skew of the matrix barcode in the field ofview of the forward-facing camera; intrinsic and extrinsic properties ofthe forward-facing camera; and the present target offset distance ortarget offset distance range for the side-facing camera, as describedabove. For example, the robotic system can: calculate a position and anorientation of a front face of the temporary shelving structure relativeto the coordinate system based on the position and the orientation ofthe matrix barcode in the field of view of the second optical sensor anda position and an orientation of the robotic system relative to thecoordinate system; and apply a target imaging distance, for theside-facing camera, from the front face of the temporary shelvingstructure to calculate the temporary waypoint.

The robotic system can queue the temporary waypoint for subsequentexecution, execute waypoints between its current location and the touchsensor, and repeat the foregoing methods during subsequent scans of theforward-facing camera to further confirm the location and orientation ofthe temporary shelving structure and to further refine the location andorientation of the temporary waypoint. The robotic system can alsoconfirm the presence, location, and orientation of the temporaryshelving structure based on data collected by the 2D or 3D depth sensor,such as based on alignment between data collected by the 2D or 3D depthsensor and a location and orientation of the temporary shelvingstructure predicted by the robotic system based on the matrix barcodedetected in images recorded by the forward-facing camera and a size andgeometry of the temporary shelving structure associated with the matrixbarcode.

Once the presence of the temporary shelving structure is confirmed, thetemporary waypoint is thus calculated, and the temporary waypoint isthus occupied by the robotic system, the robotic system can record animage of the temporary shelving structure through the side-facingcamera.

The system (e.g., the robotic system locally or the computer systemremotely) can then process this image of the temporary shelvingstructure. For example, the system can: access a list of productsassigned to the temporary shelving structure, such as by a planogramuniquely associated with the temporary shelving structure through thematrix barcode; retrieve a set of template images of products in thislist; and then compare these template images to regions in the image todetermine a current stocking condition of the temporary shelvingstructure, including a quantity of facings of these products occupyingthe temporary shelving structure, as in Block S190 described above andin U.S. patent application Ser. No. 15/600,527.

By then comparing the current stocking condition of the temporaryshelving structure to the planogram uniquely associated with thetemporary shelving structure, the system can identify deviations at thetemporary shelving structure from its target stocking condition definedby the planogram and then generate a temporary restocking listaccordingly. In particular, the system can generate a temporaryrestocking list for the temporary shelving structure based on deviationsbetween units (or facings) of products detected on the temporaryshelving structure shown in the image recorded by the robotic system atthe temporary waypoint and the planogram associated with the temporaryshelving structure. Based on the planogram of the temporary shelvingstructure and/or the planogram of the store and the location of thetemporary shelving structure determined by the robotic system during theimaging routine, the system can also: determine whether the temporaryshelving structure is properly oriented in the store, such as whetherthe front face of the temporary shelving structure is facing into itsadjacent aisle or tipped on its side; and generate a prompt to reorientor relocate the temporary shelving structure accordingly.

The system can then serve the temporary restocking list and/or theprompt to reorient or relocate the temporary shelving structure to anaffiliate of the temporary shelving structure, such by serving thetemporary restocking list and/or the prompt to a local representative orlocal restocking agency outside of the store in real-time via anelectronic notification or through a user portal. In this variation, thesystem can: serve this temporary restocking list and/or the prompt forthe temporary shelving structure to an affiliate of the temporaryshelving structure outside of the store; while also generating a globalrestocking list for other (more permanent or immutable) shelvingstructures arranged in the store and transmitting this global restockinglist to a manager or other associate of the store, as described above.

However, the system (e.g., the robotic system and/or the remote computersystem) can implement any other methods or techniques to detect, image,and monitor a temporary shelving structure arranged in the storesubstantially in real-time or asynchronously during an imaging routinein Block S164.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any suitable combination thereof. Othersystems and methods of the embodiment can be embodied and/or implementedat least in part as a machine configured to receive a computer-readablemedium storing computer-readable instructions. The instructions can beexecuted by computer-executable components integrated bycomputer-executable components integrated with apparatuses and networksof the type described above. The computer-readable medium can be storedon any suitable computer readable media such as RAMs, ROMs, flashmemory, EEPROMs, optical devices (CD or DVD), hard drives, floppydrives, or any suitable device. The computer-executable component can bea processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A method for automatically tracking inventory within a store comprising: accessing a set of images captured by a robotic system while navigating through the store during a first scan cycle; for a first image in the set of images: detecting a first set of features in the first image; detecting a first optical identifier, applied to a first shelving structure located at a first location in the store, in the first image; based on the first optical identifier, identifying a first product assigned to the first shelving structure; and detecting presence of units of the first product on the first shelving structure based on correlation between features in the first set of features detected in the first image and template visual characteristics, stored in a database, representing the first product; for a second image in the set of images: detecting a second optical identifier, applied to a second shelving structure, in the second image; based on the second optical identifier: identifying the second shelving structure as a temporary mobile inventory structure; and identifying a second product assigned to the second shelving structure; detecting a second location of the second shelving structure within the store; and generating a stock record indicating presence of units of the first product at the first location of the first shelving structure and presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 2. The method of claim 1: further comprising: detecting a second orientation of the second shelving structure within the store based on a position of the second optical identifier within the second image; defining a temporary waypoint proximal the second shelving structure based on the second location of the second shelving structure and the second orientation of the second shelving structure; at the robotic system during the first scan cycle: navigating to the temporary waypoint; and capturing a third image, in the set of images, while occupying the temporary waypoint; detecting a third set of features in the third image; and detecting presence of units of the second product on the second shelving structure based on correlation between features in the third set of features detected in the third image and template visual characteristics, stored in the database, representing the second product; and wherein generating the stock record comprises, in response to detecting presence of units of the second product on the second shelving structure, generating the stock record confirming presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 3. The method of claim 2, wherein defining the temporary waypoint proximal the second shelving structure comprises, based on the second location of the second shelving structure and the second orientation of the second shelving structure, calculating a temporary waypoint location and a temporary waypoint orientation of that, when occupied by the robotic system, locates a front face of the second shelving structure in a field of view of a camera on a side of the robotic system; and at a preset target offset distance from the camera.
 4. The method of claim 2, further comprising: prior to the first scan cycle: accessing a map of a floor space within the store generated based on map data collected by the robotic system while navigating through the store during a mapping routine preceding the first scan cycle; accessing an architectural plan defining target locations of a set of shelving structures within the store; distorting features in the architectural plan into alignment with like features in the map to generate a normalized metaspace representing real locations of the set of shelving structures; identifying the set of shelving structures within the normalized metaspace based on target locations of the set of shelving structures defined in the architectural plan and distorted into alignment with like features in the map; and defining a route along the set of shelving structures identified within the normalized metaspace, the route comprising a first route segment along a first shelving structure, in the set of shelving structures, represented in the normalized metaspace; temporarily appending the temporary waypoint to the route; and dispatching the robotic system to scan shelving structures in the set of shelving structures in the store while traversing the route during a third scan cycle succeeding the second scan cycle.
 5. The method of claim 1: further comprising: detecting a second set of features in the second image; and detecting presence of units of the second product on the second shelving structure based on correlation between features in the second set of features detected in the second image and template visual characteristics, stored in the database, representing the second product; and wherein generating the stock record comprises, in response to detecting presence of units of the second product on the second shelving structure, generating the stock record confirming presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 6. The method of claim 1: further comprising detecting a second orientation of the second shelving structure within the store; and wherein generating the stock record comprises generating the stock record further indicating presence of units of the second product in the temporary mobile inventory structure at the second orientation during the first scan cycle.
 7. The method of claim 6, wherein detecting the second location of the second shelving structure within the store and detecting the second orientation of the second shelving structure within the store comprise: extracting an image position and an image orientation of the second optical identifier from the second image; accessing a robotic system position and a robotic system orientation of the robotic system within the store during recordation of the store during recordation of the second image; and calculating the second location and the second orientation of a front face of the second shelving structure within the store based on the image position, the image orientation, the robotic system position, and robotic system orientation.
 8. The method of claim 7, further comprising: identifying an aisle in the store occupied by the second shelving structure based on the second location of the second shelving structure; detecting misorientation of the shelving structure within the store based on a difference between the second orientation of the front face of the second shelving structure and the aisle; and in response to detecting misorientation of the second shelving structure, generating a prompt to reorient the second shelving structure within the aisle.
 9. The method of claim 1, further comprising: identifying an aisle in the store occupied by the second shelving structure based on the second location of the second shelving structure; detecting displacement of the shelving structure within the aisle based on the second location of the second shelving structure; and in response to detecting displacement of the second shelving structure within the aisle, generating a prompt to relocate the second shelving structure within the aisle.
 10. The method of claim 1: wherein detecting the second optical identifier comprises detecting the second optical identifier comprising a matrix barcode, applied to the second shelving structure, in the second image; and wherein identifying the second shelving structure as the temporary mobile inventory structure comprises identifying the second shelving structure as the temporary mobile inventory structure based on the matrix barcode.
 11. The method of claim 1: wherein identifying the second shelving structure as the temporary mobile inventory structure comprises, based on the second optical identifier, identifying the second shelving structure as a freestanding cardboard popup associated with a particular brand; and wherein identifying the second product assigned to the second shelving structure comprises, based on the second optical identifier, identifying the second product associated with the particular brand and designated for the freestanding cardboard popup.
 12. The method of claim 1: wherein identifying the second product assigned to the second shelving structure comprises, based on the second optical identifier, retrieving a second list of products assigned to the second shelving structure; further comprising: retrieving a second set of template images from a database of template images, each template image in the second set of template images comprising visual features of a product in the second list of products; and confirming presence of a unit of the second product, in the second list of products, on the second shelving structure in response to correlation between features in the second image and features in a first template image, in the second set of template images, representing the second product; and wherein generating the stock record comprises, in response to confirming presence of the unit of the second product, generating the stock record indicating presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 13. The method of claim 1: wherein identifying the second product assigned to the second shelving structure comprises, based on the second optical identifier, retrieving a second list of products assigned to the second shelving structure; further comprising: retrieving a second set of template images from a database of template images, each template image in the second set of template images comprising visual features of a product in the second list of products; generating a count of units of the second product, in the second list of products, on the second shelving structure based on correlations between features in the second image and features in the second set of template images representing the second product; detecting incomplete stocking of the second shelving structure in response to deviation between the count of units of the second product and a target quantity of units of the second product assigned to the second shelving structure; and in response to detecting incomplete stocking of the second shelving structure, generating a temporary restocking prompt for the second shelving structure; and wherein generating the stock record comprises generating the stock record indicating presence of units of the first product at the first location of the first shelving structure and the count of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 14. A method for automatically tracking inventory within a store comprising: accessing a set of images captured by a robotic system while navigating through the store during a first scan cycle; for a first image in the set of images: detecting a first set of features in the first image; detecting a first optical identifier, applied to a first shelving structure located at a first location in the store, in the first image; based on the first optical identifier, identifying a first product assigned to the first shelving structure; and detecting presence of units of the first product on the first shelving structure based on correlation between features in the first set of features detected in the first image and template visual characteristics, stored in a database, representing the first product; for a second image in the set of images: detecting a second optical identifier, applied to a second shelving structure, in the second image; detecting a second location of the second shelving structure within the store; identifying the second shelving structure as a temporary mobile inventory structure based on the second location of the second shelving structure intersecting an aisle of the store; and based on the second optical identifier, identifying a second product assigned to the second shelving structure; and generating a stock record indicating presence of units of the first product at the first location of the first shelving structure and presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 15. The method of claim 14: further comprising: detecting a second orientation of the second shelving structure within the store based on a position of the second optical identifier within the second image; defining a temporary waypoint proximal the second shelving structure based on the second location of the second shelving structure and the second orientation of the second shelving structure; at the robotic system: navigating to the temporary waypoint; and capturing a third image, in the set of images, while occupying the temporary waypoint; detecting a third set of features in the third image; and detecting presence of units of the second product on the second shelving structure based on correlation between features in the third set of features detected in the third image and template visual characteristics, stored in the database, representing the second product; and wherein generating the stock record comprises, in response to detecting presence of units of the second product on the second shelving structure, generating the stock record confirming presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 16. The method of claim 14: further comprising: detecting a second set of features in the second image; and confirming presence of units of the second product on the second shelving structure based on correlation between features in the second set of features detected in the second image and template visual characteristics, stored in the database, representing the second product; and wherein generating the stock record comprises, in response to confirming presence of units of the second product on the second shelving structure, generating the stock record indicating presence of units of the second product in the temporary mobile inventory structure at the second location during the first scan cycle.
 17. The method of claim 14: further comprising detecting a second optical identifier, applied to the second shelving structure, in the second image; and wherein identifying the second shelving structure as the temporary mobile inventory structure comprises identifying the second shelving structure as the temporary mobile inventory structure further based on the second optical identifier.
 18. A method for automatically mapping shelving structures within a store comprising: accessing a set of images captured by a camera system deployed in the store; for a first image in the set of images: detecting a first optical identifier, applied to a first shelving structure in a set of fixed shelving structures present in the store, in the first image; based on the first optical identifier, identifying a first product assigned to the first shelving structure; detecting presence of a unit of the first product on the first shelving structure based on correlation between features in the first set of features detected in the first image and template visual characteristics, stored in a database, representing the first product; and calculating a first stock condition, of the first shelving structure, indicating presence of the first unit of the first product on the first shelving structure; for a second image in the set of images: detecting a second optical identifier, applied to a second shelving structure, in the second image; based on the second optical identifier: identifying the second shelving structure as a temporary mobile inventory structure based on the second optical identifier; and identifying a second product assigned to the second shelving structure; confirming presence of a second unit of the second product on the second shelving structure based on correlation between features detected in the second image and template visual characteristics, stored in the database, representing the second product; and generating a second stock condition, of the second shelving structure, indicating presence of the second unit of the second product on the second shelving structure; and aggregating the first stock condition of the first shelving structure and the second stock condition of the second shelving structure into a stock record of the store.
 19. The method of claim 18: further comprising dispatching a mobile robotic system, comprising a set of optical sensors defining the camera system, to autonomously navigate through the store and capture the set of images of inventory structures within the store during a scan cycle; and wherein accessing the set of images comprises accessing the set of images captured by the mobile robotic system during the scan cycle.
 20. The method of claim 18: further comprising detecting a second orientation of the second shelving structure within the store; and wherein generating the stock record comprises generating the stock record further indicating presence of units of the second product in the temporary mobile inventory structure at the second orientation during the first scan cycle. 